Grouping and ranking images based on facial recognition data

ABSTRACT

Technologies for grouping images, and ranking the images and the groupings, based on entities shown in the images. Images may be grouped based on faces shown in the images. Different images with faces that indicate the same entity (e.g., Adam) may be automatically grouped together. Different images with faces that indicate the same multiple entities (e.g., the people in my family) may also be automatically grouped together. Such automatic grouping may be based on facial recognition technologies. Further, images and groups of images may be automatically ranked based on the faces shown and entities represented. Such rankings may also be influenced by adjacent data that indicates family and friends and the like, and that can be used to identify such entities in the images.

BACKGROUND

Thanks to advances in imaging technologies, people take more picturesthan ever before. Further, the proliferation of media sharingapplications has increased the demand for picture sharing to a greaterdegree than ever before. Yet the flood of photos, and the need to sortthrough them to find relevant pictures, has actually increased the timeand effort required for sharing pictures. As a result, it is often thecase that either pictures that are less than representative of the bestpictures, or no pictures at all, end up getting shared.

Most people have many pictures or videos that include different people(or other entities) including family, friends, acquaintances, andstrangers. In many cases, not all entities are treated equally whendeciding the importance of image. For example, images that show familyand friends are typically treated as more important than images thatshow mostly strangers or the like. But sorting through today's largenumber of images to select the more important ones can be prohibitivelytime-consuming.

SUMMARY

The summary provided in this section summarizes one or more partial orcomplete example embodiments of the invention in order to provide abasic high-level understanding to the reader. This summary is not anextensive description of the invention and it may not identify keyelements or aspects of the invention, or delineate the scope of theinvention. Its sole purpose is to present various aspects of theinvention in a simplified form as a prelude to the detailed descriptionprovided below.

The invention encompasses technologies for grouping images, and rankingthe images and the groupings, based on entities shown in the images.Images may be grouped based on faces shown in the images. Differentimages with faces that indicate the same entity (e.g., Adam) may beautomatically grouped together. Different images with faces thatindicate the same multiple entities (e.g., the people in my family) mayalso be automatically grouped together. Such automatic grouping may bebased on facial recognition technologies. Further, images and groups ofimages may be automatically ranked based on the faces shown and entitiesrepresented. Such rankings may also be influenced by adjacent data thatindicates family and friends and the like, and that can be used toidentify such entities in the images.

Many of the attendant features will be more readily appreciated as thesame become better understood by reference to the detailed descriptionprovided below in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

The detailed description provided below will be better understood whenconsidered in connection with the accompanying drawings, where:

FIG. 1 is a block diagram showing an example computing environment inwhich the invention described herein may be implemented.

FIG. 2 is a block diagram showing an example system configured forrecognizing faces in a set of images, and grouping and ranking theimages and/or the groupings of the images.

FIG. 3 is a block diagram showing an example method for grouping images.

FIG. 4 is a block diagram showing an example method for ranking imagesand/or groups of images.

Like-numbered labels in different figures are used to designate similaror identical elements or steps in the accompanying drawings.

DETAILED DESCRIPTION

The detailed description provided in this section, in connection withthe accompanying drawings, describes one or more partial or completeexample embodiments of the invention, but is not intended to describeall possible embodiments of the invention. This detailed descriptionsets forth various examples of at least some of the technologies,systems, and/or methods invention. However, the same or equivalenttechnologies, systems, and/or methods may be realized according toexamples as well.

Although the examples provided herein are described and illustrated asbeing implementable in a computing environment, the environmentdescribed is provided only as an example and not a limitation. As thoseskilled in the art will appreciate, the examples disclosed are suitablefor implementation in a wide variety of different computingenvironments.

FIG. 1 is a block diagram showing an example computing environment 100in which the invention described herein may be implemented. A suitablecomputing environment may be implemented with numerous general purposeor special purpose systems. Examples of well known systems include, butare not limited to, cell phones, personal digital assistants (“PDA”),personal computers (“PC”), hand-held or laptop devices,microprocessor-based systems, multiprocessor systems, systems on a chip(“SOC”), servers, Internet services, workstations, consumer electronicdevices, cell phones, set-top boxes, and the like. In all cases, suchsystems are strictly limited to articles of manufacture and the like.

Computing environment 100 typically includes a general-purpose computingsystem in the form of a computing device 101 coupled to variouscomponents, such as peripheral devices 102, 103, 101 and the like. Thesemay include components such as input devices 103, including voicerecognition technologies, touch pads, buttons, keyboards and/or pointingdevices, such as a mouse or trackball, that may operate via one or moreinput/output (“I/O”) interfaces 112. The components of computing device101 may include one or more processors (including central processingunits (“CPU”), graphics processing units (“GPU”), microprocessors(“μP”), and the like) 107, system memory 109, and a system bus 108 thattypically couples the various components. Processor(s) 107 typicallyprocesses or executes various computer-executable instructions and,based on those instructions, controls the operation of computing device101. This may include the computing device 101 communicating with otherelectronic and/or computing devices, systems or environments (not shown)via various communications technologies such as a network connection 114or the like. System bus 108 represents any number of bus structures,including a memory bus or memory controller, a peripheral bus, a serialbus, an accelerated graphics port, a processor or local bus using any ofa variety of bus architectures, and the like.

System memory 109 may include computer-readable media in the form ofvolatile memory, such as random access memory (“RAM”), and/ornon-volatile memory, such as read only memory (“ROM”) or flash memory(“FLASH”). A basic input/output system (“BIOS”) may be stored innon-volatile or the like. System memory 109 typically stores data,computer-executable instructions and/or program modules comprisingcomputer-executable instructions that are immediately accessible toand/or presently operated on by one or more of the processors 107.

Mass storage devices 104 and 110 may be coupled to computing device 101or incorporated into computing device 101 via coupling to the systembus. Such mass storage devices 104 and 110 may include non-volatile RAM,a magnetic disk drive which reads from and/or writes to a removable,non-volatile magnetic disk (e.g., a “floppy disk”) 105, and/or anoptical disk drive that reads from and/or writes to a non-volatileoptical disk such as a CD ROM, DVD ROM 106. Alternatively, a massstorage device, such as hard disk 110, may include non-removable storagemedium. Other mass storage devices may include memory cards, memorysticks, tape storage devices, and the like.

Any number of computer programs, files, data structures, and the likemay be stored in mass storage 110, other storage devices 104, 105, 106and system memory 109 (typically limited by available space) including,by way of example and not limitation, operating systems, applicationprograms, data files, directory structures, computer-executableinstructions, and the like.

Output components or devices, such as display device 102, may be coupledto computing device 101, typically via an interface such as a displayadapter 111. Output device 102 may be a liquid crystal display (“LCD”).Other example output devices may include printers, audio outputs, voiceoutputs, cathode ray tube (“CRT”) displays, tactile devices or othersensory output mechanisms, or the like. Output devices may enablecomputing device 101 to interact with human operators or other machines,systems, computing environments, or the like. A user may interface withcomputing environment 100 via any number of different I/O devices 103such as a touch pad, buttons, keyboard, mouse, joystick, game pad, dataport, and the like. These and other I/O devices may be coupled toprocessor 107 via I/O interfaces 112 which may be coupled to system bus108, and/or may be coupled by other interfaces and bus structures, suchas a parallel port, game port, universal serial bus (“USB”), fire wire,infrared (“IR”) port, and the like.

Computing device 101 may operate in a networked environment viacommunications connections to one or more remote computing devicesthrough one or more cellular networks, wireless networks, local areanetworks (“LAN”), wide area networks (“WAN”), storage area networks(“SAN”), the Internet, radio links, optical links and the like.Computing device 101 may be coupled to a network via network adapter 113or the like, or, alternatively, via a modem, digital subscriber line(“DSL”) link, integrated services digital network (“ISDN”) link,Internet link, wireless link, or the like.

Communications connection 114, such as a network connection, typicallyprovides a coupling to communications media, such as a network.Communications media typically provide computer-readable andcomputer-executable instructions, data structures, files, programmodules and other data using a modulated data signal, such as a carrierwave or other transport mechanism. The term “modulated data signal”typically means a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communications media may includewired media, such as a wired network or direct-wired connection or thelike, and wireless media, such as acoustic, radio frequency, infrared,or other wireless communications mechanisms.

Power source 190, such as a battery or a power supply, typicallyprovides power for portions or all of computing environment 100. In thecase of the computing environment 100 being a mobile device or portabledevice or the like, power source 190 may be a battery. Alternatively, inthe case computing environment 100 is a desktop computer or server orthe like, power source 190 may be a power supply designed to connect toan alternating current (“AC”) source, such as via a wall outlet.

Some mobile devices may not include many of the components described inconnection with FIG. 1. For example, an electronic badge may becomprised of a coil of wire along with a simple processing unit 107 orthe like, the coil configured to act as power source 190 when inproximity to a card reader device or the like. Such a coil may also beconfigure to act as an antenna coupled to the processing unit 107 or thelike, the coil antenna capable of providing a form of communicationbetween the electronic badge and the card reader device. Suchcommunication may not involve networking, but may alternatively begeneral or special purpose communications via telemetry, point-to-point,RF, IR, audio, or other means. An electronic card may not includedisplay 102, I/O device 103, or many of the other components describedin connection with FIG. 1. Other mobile devices that may not includemany of the components described in connection with FIG. 1, by way ofexample and not limitation, include electronic bracelets, electronictags, implantable devices, and the like.

Those skilled in the art will realize that storage devices utilized toprovide computer-readable and computer-executable instructions and datacan be distributed over a network. For example, a remote computer orstorage device may store computer-readable and computer-executableinstructions in the form of software applications and data. A localcomputer may access the remote computer or storage device via thenetwork and download part or all of a software application or data andmay execute any computer-executable instructions. Alternatively, thelocal computer may download pieces of the software or data as needed, ordistributively process the software by executing some of theinstructions at the local computer and some at remote computers and/ordevices.

Those skilled in the art will also realize that, by utilizingconventional techniques, all or portions of the software'scomputer-executable instructions may be carried out by a dedicatedelectronic circuit such as a digital signal processor (“DSP”),programmable logic array (“PLA”), discrete circuits, and the like. Theterm “electronic apparatus” may include computing devices or consumerelectronic devices comprising any software, firmware or the like, orelectronic devices or circuits comprising no software, firmware or thelike.

The term “firmware” typically refers to executable instructions, code,data, applications, programs, program modules, or the like maintained inan electronic device such as a ROM. The term “software” generally refersto computer-executable instructions, code, data, applications, programs,program modules, or the like maintained in or on any form or type ofcomputer-readable media that is configured for storingcomputer-executable instructions or the like in a manner that isaccessible to a computing device. The term “computer-readable media” andthe like as used herein is strictly limited to one or more apparatus,article of manufacture, or the like that is not a signal or carrier waveper se. The term “computing device” as used in the claims refers to oneor more devices such as computing device 101 and encompasses clientdevices, mobile devices, one or more servers, network services such asan Internet service or corporate network service, and the like, and anycombination of such.

FIG. 2 is a block diagram showing an example system 200 configured forrecognizing faces in a set of images, and grouping and ranking theimages and/or the groupings of the mages. Such rankings may be based ona frequency of the faces and/or face signatures. The system may includeseveral modules such as facial recognition engine 210 that accepts input211 and provides output 212, data store 220, grouping engine 230 thataccepts input 212 and produces output 232, and ranking engine 240 thataccepts input 212 and produces output 242. Each of these modules(including any sub-modules) may be implemented in hardware, firmware,software (e.g., as program modules comprising computer-executableinstructions), or any combination thereof. Each such module may beimplemented on/by one device, such as a computing device, or acrossmultiple such devices. For example, one module may be implemented in adistributed fashion on/by multiple devices such as servers or elementsof a network service or the like. Further, each such module (includingany sub-modules) may encompass one or more sub-modules or the like, andthe modules may be implemented as separate modules, or any two or moremay be combined in whole or in part. The division of modules (includingany sub-modules) described herein in non-limiting and intended primarilyto aid in describing aspects of the invention. The phrase “face in theimage” and the like refers not to an actual face in an image, but to animage or representation of a face, actual or otherwise.

In summary, system 200 is configured for detecting faces in input images211, generating a face identifier for each detected face in the set ofimages, grouping images that include faces of the same entity, andranking images and/or groups of images based on the faces detected inthe images. A set of images is typically provided by one or more sourcesas input 212 to the system. Such sources include camera phones, digitalcameras, digital video recorders (“DVRs”), computers, digital photoalbums, social media applications, image and video streaming web sites,and any other source of digital images. Note that one or more actualimages may be input, or references to images, or any combination ofsuch. Further, the phrase “grouping images”, adding an image to agroup”, and the like as used herein include grouping actual images,grouping references to images, and/or any combination of the foregoing.

Facial recognition engine 210 is a module that accepts an image as input212, detects one or more faces in the image, and detects variousfeatures in recognized faces. In one example, the functionality ofmodule 210 may be provided in the form of a software development kit(“SDK”). In one example, facial recognition engine 210 may providefacial recognition data as one or more outputs, each of which may bestored in data store 220. One output may be in the form of a faceidentifier that identifies a detected face in an image 212. Givenmultiple detected faces in an image, a unique face identifier istypically provided for each face detected in the image. In one example,a face identifier may be a RECT data structure or the like that boundscertain aspects of the face it identifies. Such a RECT data structuremay indicate a position in the image of the face it identifies, and/ormay indicate a size or relative size of the identified face in theimage. Any face identifier(s) that are output 212 may be accepted asinput by data store 220, grouping engine 230, and/or ranking engine 240.

Another output 212 of facial recognition engine 210 may be in the formof a set of facial feature descriptors that describe facial featuresdetected in a face corresponding to the face's identifier. Givenmultiple face identifiers as input, a corresponding set of facialfeature descriptors is typically provided for each face identifier. Inone example, the set of facial feature descriptors may be in the form ofcoordinates for each detected facial feature, such as the eyes,eyebrows, nose, and mouth of the face.

Another output 212 of facial recognition engine 210 may be in the formof a face score corresponding to a face identifier. Such a face scoremay be an overall quality score for the face that is based on facialfeature analysis. In one example, the score may be a value between zeroand one. In other examples, the score may be represented by a valuewithin a continuous range, or by a quantization such as high, medium, orlow, or the like. In one example, the face score may represent anoverall measure of the quality of the face in the image, and may bebased on a combination of analyzed aspects such as face sharpness, faceexpression, face pose, proximity to image edges, open/closed state ofthe face's eyes and mouth, and/or other aspects.

Another output 212 of facial recognition engine 210 may be in the formof a face signature that, across the images in the set, uniquelyidentifies an entity that the face represents, at least within the scopeof the detected features. For example, if various face shots of Adamappear in several images in a set, then each of Adam's face shots willhave the same face signature that uniquely identifies the entity “Adam”,at least within the scope of the detected features. Such a facesignature is typically based on analysis of a face's identifier andcorresponding set of facial feature descriptors. Such face signaturesmay be used to determine other faces in other images of the set thatrepresent the same entity, and thus may be used to determine a frequencythat a particular entity appears in the image set. The term “sameentity” as used herein typically refers to a particular entity (e.g., aperson).

Another output 212 of facial recognition engine 210 may be the set ofimages provided as input 211. One or more of these images, and/orreferences to them, may be stored in data store 220. Further, suchimages may also be retrieved as input 211 to facial recognition engine210, as may any other of the outputs 212 stored in data store 220. Notethat the movement of an image described herein, such as providing,retrieving, or the like, refers to movement of the actual image itselfand/or to a reference to the actual image.

One example of facial recognition engine 210 is provided as system 200described in US patent application Ser. No. 14/266,795, filed on Apr.30, 2014, and entitled “RATING PHOTOS FOR TASKS BASED ON CONTENT ANDADJACENT SIGNALS”, that is incorporated herein by reference in itsentirety.

Data store 220 is a persistent data store such as one or more databaseand/or other storage system. Data store 220 may be integral to system200 or may be separate. Further, adjacent information input 221 fromadjacent information sources may be provided to system 200 via datastore 220, or may be provided directly to system 200 without necessarilybeing stored in data store 220. Adjacent information may be obtainedfrom sources that are generally unrelated or indirectly related to theimages in the set. In general, any system or data source that can beaccessed by system 200 may be an adjacent information source.Non-limiting examples of adjacent information sources include calendars,social media applications, news sources, blogs, email, location trackinginformation, and any other source.

Grouping engine 230 is a module that accepts one or more of the outputsof facial recognition engine 210 directly and/or from data store 220.Grouping engine 230 may group images that include one or more faces withthe same face signature. For example, if the first and third images in aset of images both include a face with the same face signature, thenthose two images may be grouped together, along with any other images inthe set that include a face with the same face signature. Thus, eachimage in the group includes at least one face that represents the sameentity as the other images in the group. Faces with a face score below acertain threshold may be excluded from the grouping as if not present inthe image in which they are detected.

Grouping engine may also analyze face information from an image, such asa face identifier and/or the corresponding set of facial featuredescriptors. The results of such analysis may be used separate from orin addition to face signatures in the grouping of images.

Groups that share faces of a single entity in common may be described assingle-entity groups. Groups that share faces of two different entitiesin common may be described as double-entity groups, and so forth. Forexample, each image in a set may each show the same family of fivepeople. In this example, all images in the set may be grouped togetherin a five-entity group because each image share five face signatures incommon, one for each member of the family. Such a five-entity group maybe described as a larger-entity group than a single-entity group or afour-entity group, and so forth.

Images without detected faces may be grouped together. Images with oneor more detected faces that do not share face signatures with faces inany other images may be grouped by themselves.

Groupings of images produced by grouping engine 230 may be provided asoutput 232. Such provided groups may comprise the images themselves, ormay be comprised of references to the images, or any combination of theforegoing. Such groupings may be automatically provided, such as beingpresented in photo albums, shared via social media applications, or thelike. Priority in presenting such groupings may be given to groupingswith larger numbers of images, and/or to larger-entity groups.

Ranking engine 240 is a module that accepts one or more of the outputsof facial recognition engine 210 directly and/or from data store 220,and/or groupings of images provided by grouping engine 230. Rankingengine 240 may rank images that include one or more faces based on facescores of the faces detected in the image. Faces with a score below acertain threshold may be excluded from the rankings as if not present inthe image in which they are detected. Such scores may be weighted toreflect the relative importance of various faces and/or face aspects inthe image. For example, faces with a higher frequency of appearance inthe set of images may be weighted higher than those with a lowerfrequency of appearance, such as determined by face signatures and/orgroupings. Further, faces of entities that are determined to be friendsor family or the like of a person providing the set of images may beweighted higher than faces of entities that are not so determined. Inone example, such a determination may be based on adjacent informationinput 221, or based on other input to system 200 such as input providedby the person or other entity.

Ranking engine 240 may also rank groupings of images based on the facesof the common entities in the images in the groupings. For example,larger-entity groupings may be ranked higher than smaller-entitygroupings. Further, groupings with a larger number of faces of entitiesthat are determined to be friends or family or the like may be rankedhigher than groupings with a lesser number of such.

Rankings of images produced by ranking engine 240 may be provided asoutput 242. Such provided rankings may comprise the images themselves,or may be comprised of references to the images, or any combination ofthe foregoing. Such rankings may be automatically provided, such asbeing presented in photo albums, shared via social media applications,or the like.

FIG. 3 is a block diagram showing an example method 300 for groupingimages. Such a method may be performed by grouping engine 230 or thelike. Block 310 typically indicates performing the steps of method 300for each image in a set of images, such as provided at input 211 or thelike. In other examples, the steps may be performed in an orderdifferent than that shown, and/or one or more steps may be omitted orcombined with other steps.

Block 320 typically indicates receiving facial recognition data for aface detected in an image. Such data may include a face identifier, aset of facial feature descriptors, and face score, and/or a facesignature. The image and/or a reference to the image may also beprovided. Once the facial recognition data is received, method 300typically continues at block 330.

Block 330 typically indicates considering a face score of the facecorresponding to the received face identifier. Such a face score may bereceived at step 320 as part of the facial recognition data. In oneexample, the face score may be weighted. If the face score, with orwithout its weight, is below a certain threshold, then the face may bedropped from grouping consideration. In this case, method 300 continuesat block 360. Otherwise, method 300 typically continues at block 340.

Block 340 typically indicates considering a face signature of the facecorresponding to the received face identifier. Such a face signature maybe received at step 320 as part of the facial recognition data. Notethat the face signature indicates an entity that the face represents. Inone example, the face signature is compared to that of any existinggroup(s). If no existing group is associated with the face signature,then a new group is created for that face signature, and the image isadded to that group (step 350). For each existing group that isassociated with the face signature, the image is added to that group(step 350). In this example, single-entity groups are created andfilled.

For example, if the face signature indicates the entity “Adam”, and nogroup for mages with faces of Adam exists, then a group is created forimages with faces of Adam, and the mage is added. If a group for facesof Adam already exists, then the image with Adam's face is added to theAdam group (step 350). The term “single-entity group” as used hereinrefers to a group of images where each image in the group includes adetected face that represents the same entity.

In another example, if the image with the face signature indicating Adamis also associated with a face signature indicating “Mary”, then theimage is also added (step 350) to any existing group for Adam and Mary,or to a newly-created group if one does not already exist, and so forthfor each additional face signature associated with the image. In thisexample, multi-entity groups are created and filled. Once the facesignature(s) is considered and the image is added to the appropriategroup(s), then the method typically continues at step 360. The term“multi-entity group” as used herein refers to an n-entity group ofimages where each image in the group includes a detected face thatrepresents each of the same n entities. In one example, given athree-entity group with several images, each image includes threedetected faces, one that represents each of Adam, Mary, and Jim.

Block 360 typically indicates determining if any more faces are presentin the image. In one example, this determination is made based on thefacial recognition data received at step 320. If there are additionalfaces in the image that have not yet been considered (e.g., based on aface identifier), then the method typically continues at step 330 forone of the yet-to-be considered faces in the image. Otherwise, themethod typically continues at step 370.

Block 370 typically indicates determining if any more images are presentin the set of images. If there are additional images in the set thathave not yet been considered, then the method typically continues atstep 310 for one of the yet-to-be considered images in the set.Otherwise, the method is typically done.

FIG. 4 is a block diagram showing an example method 400 for rankingimages and/or groups of images. Such a method may be performed byranking engine 240 or the like. Block 410 typically indicates performingthe steps of method 400 for each face detected in an image and/or a setof images, such as provided at input 211 or the like. In other examples,the steps may be performed in an order different than that shown, and/orone or more steps may be omitted or combined with other steps.

Block 420 typically indicates receiving facial recognition data for aface detected in an image. Such data may include a face identifier, aset of facial feature descriptors, and face score, and/or a facesignature. The image and/or a reference to the image may also beprovided. Once the facial recognition data is received, method 400typically continues at block 430.

Block 430 typically indicates considering a face score of the facecorresponding to the received face identifier. Such a face score may bereceived at step 420 as part of the facial recognition data. In oneexample, the face score may be weighted. If the face score, with orwithout its weight, is below a certain threshold, then the face may bedropped from ranking consideration. In this case, method 400 continuesat block 460. Otherwise, method 400 typically continues at block 440.

Block 440 typically indicates considering a face signature of the facecorresponding to the received face identifier. Such a face signature maybe received at step 420 as part of the facial recognition data. Notethat the face signature indicates an entity that the face represents.The face signature may be used to determine a frequency at which facesof a particular entity appears in an image or group of images. In oneexample, the face signature of an entity is associated with a count ofdetected faces in an image and/or a group of images that correspond tothe entity. Facial recognition engine 210 may provide such a count giventhe face signature and an image, set of images, and/or grouping ofimages, or references thereto.

For example, if the face signature indicates the entity “Adam”, and theface signature is associated with a larger number of detected faces inthe image and/or group of images, then the image and/or group may beranked higher than if the face signature is associated with a lessernumber of such detected faces (step 450).

In another example, if the image with the face signature indicating Adamis also associated with a face signature indicating “Mary”, then theimage is a multi-entity image. In this example, multi-entity images withdetected faces of a larger number of entities may be ranked higher thanmulti-entity images with a lesser number of such detected faces (step450). In a similar fashion, multi-entity groups with a larger number ofentities may be ranked higher than multi-entity groups with a lessernumber of entities.

In yet another example, the rankings may be weighted by or based on theface signature indicating an entity determined to be a friend or familyor the like. In this example, images and/or groups of images with alarger number of friends or family or the like may be ranked higher thatimages and/or groups of images with a lesser number of such.

The ranking of images or groups of images may be in the form of arelative rank between the images or groups. In another example, theranking may be in the form of a score or priority assigned to each imageor group.

Once the face signature is considered, then the method typicallycontinues at step 460. Further, rankings may be finalized after alldetected faces in an image and/or group of images, or all facesignatures associated with an image and/or group of images, have beenprocessed. For example, ranking information may have been accumulatedfor each considered face and/or each considered face signature. Suchaccumulated rankings may then be consolidated into a single ranking foran image and/or group of images. In one example, a sum or product ofsuch accumulated rankings may be assigned as a finalized ranking to animage and/or group of images.

Block 460 typically indicates determining if any more faces are presentin the image. In one example, this determination is made based on thefacial recognition data received at step 420. If there are additionalfaces in the image that have not yet been considered (e.g., based on aface identifier), then the method typically continues at step 430 forone of the yet-to-be considered faces in the image. Otherwise, themethod typically continues at step 470.

Block 470 typically indicates determining if any more images need to beconsidered, such as when processing a group of images as opposed to anindividual image. If there are additional images that have not yet beenconsidered, then the method typically continues at step 410 for one ofthe yet-to-be considered images. Otherwise, the method is typicallydone.

In view of the many possible embodiments to which the invention and theforgoing examples may be applied, it should be recognized that theexamples described herein are meant to be illustrative only and shouldnot be taken as limiting the scope of the present invention. Therefore,the invention as described herein contemplates all such embodiments asmay come within the scope of the following claims and any equivalentsthereto.

The invention claimed is:
 1. A method performed on a computing device,the method comprising: determining, by the computing device for eachimage in a set of images, facial recognition data that, for each facedetected by the computing device in the each image, comprises a faceidentifier that uniquely identifies the each face, a set of facialfeature descriptors, a face score that is based on an open/closed stateof the each face's eyes and mouth and that indicates an overall qualityof the each face, and a face signature that, across the images in theset, uniquely identifies an entity that the each face represents; andgrouping, by the computing device based at least on the face signaturesand the face scores, images of the set into one or more groups, whereeach group comprises one or more images that each show a detected facethat represents a same entity as that represented by any other detectedface shown in any image in the each group.
 2. The method of claim 1where the grouping is further based on the face scores.
 3. The method ofclaim 1 where a face with a face score below a particular threshold isnot included in the grouping.
 4. The method of claim 1 where at leastone of the groups is a multi-entity group.
 5. The method of claim 1further comprising ranking, based on the face scores, the images in theset or the groups.
 6. The method of claim 1 further comprising rankingthe groups according to a number of entities of the groups.
 7. Themethod of claim 1 further comprising ranking the groups according toentities of the groups matching adjacent data that indicates family andfriends.
 8. A system comprising a computing device and at least oneprogram module that together are configured for performed actionscomprising: determining, by the computing device for each image in a setof images, facial recognition data that, for each face detected by thecomputing device in the each image, comprises a face identifier thatuniquely identifies the each face, a set of facial feature descriptors,a face score that is based on an open/closed state of the each face'seyes and mouth and that indicates an overall quality of the face, and aface signature that, across the images in the set, uniquely identifiesan entity that the each face represents; and grouping, by the computingdevice based at least on the face signatures and the face scores, imagesof the set into one or more groups, where each group comprises one ormore images that each show a detected face that represents a same entityas that represented by any other detected face shown in any image in theeach group.
 9. The system of claim 8 where the grouping is further basedon the face scores.
 10. The system of claim 8 where a face with a facescore below a particular threshold is not included in the grouping. 11.The system of claim 8 where at least one of the groups is a multi-entitygroup.
 12. The system of claim 8, the actions further comprisingranking, based on the face scores, the images in the set or the groups.13. The system of claim 8, the actions further comprising ranking thegroups according to a number of entities of the groups.
 14. The systemof claim 8, the actions further comprising ranking the groups accordingto entities of the groups matching adjacent data that indicates familyand friends.
 15. At least one computer-readable media storingcomputer-executable instructions that, when executed by a computingdevice, cause the computing device to perform actions comprising:determining, by the computing device for each image in a set of images,facial recognition data that, for each face detected by the computingdevice in the image, comprises a face identifier that uniquelyidentifies the each face, a set of facial feature descriptors, a facescore that is based on an open/closed state of the each face's eyes andmouth and that indicates an overall quality of the each face, and a facesignature that, across the images in the set, uniquely identifies anentity that the each face represents; and grouping, by the computingdevice based at least on the face signatures and the face scores, imagesof the set into one or more groups, where each group comprises one ormore images that each show a detected face that represents a same entityas that represented by any other detected face shown in any image in theeach group.
 16. The at least one computer-readable media of claim 15where the grouping is further based on the face scores, or where a facewith a face score below a particular threshold is not included in thegrouping.
 17. The at least one computer-readable media of claim 15 whereat least one of the groups is a multi-entity group.
 18. The at least onecomputer-readable media of claim 15, the actions further comprisingranking, based on the face scores, the images in the set or the groups.19. The at least one computer-readable media of claim 15, the actionsfurther comprising ranking the groups according to a number of entitiesof the groups.
 20. The at least one computer-readable media of claim 15,the actions further comprising ranking the groups according to entitiesof the groups matching adjacent data that indicates family and friends.